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ABSTRACT 

^  By  perturbing  properly  a  linear  program  to  a  separable  quadratic  program 
it  Is  possible  to  solve  the  latter  In  Its  dual  variable  space  by  Iterative 
techniques  such  as  sparsity-preserving  SOR  (successive  overrelaxatlon) 
algorithms.  The  main  result  of  this  paper  gives  an  effective  computational 
criterion  to  check  whether  the  solutions  of  the  perturbed  quadratic  programs 
provide  the  least  2-norm  solution  of  the  original  linear  program. 


AMS  (MOS)  Subject  Classifications:  90C05,  90C20 

Key  Words:  Linear  programming.  Quadratic  programming 

Work  unit  Number  5  -  Optimization  and  Large  Scale  Systems 


SIGNIFICANCE  AND  BXFIAMATION 


A  novel  way  la  presented  In  which  the  smallest  solution  of  a  linear 
programming  problem  can  be  determined.  This  result  leads  to  an  effective 
computational  way  for  solving  very  large  sparse  linear  programs. 


A  MEW  MESUI,T  IN  THE  THEORY  AMD  COMPUTATION  OF  THE 
LEAST  NORM  SOLUTION  OF  A  LINEAR  PROGRAM 


St«fano  Lucldi 

1 .  INTRODUCTION 

Recently,  it  has  been  recognized  that  one  of  the  most  promising  approach  for  solving 
very  large  linear  programs  is  based  on  iterative  SOR  (successive  overrelaxation)  methods- 

It  was  shown  in  [1]  and  [2]  that  the  least  2-norm  solution  of  a  linear  program  can  be 
obtained  by  perturbing  "properly*  the  linear  program  to  a  separable  quadratic  program  and 
by  solving  the  latter  in  its  dual  variable  space  by  iterative  techniques  such  as  SOR 
methods . 

In  this  context  some  very  effective  algorithms  were  proposed  in  (3) .  The  principal 
and  computationally-distinguishing  features  of  these  SOR  algorithms  are  that  they  preserve 
the  sparsity  structure  of  the  problem  and  require  only  simple  operations,  and,  hence,  very 
large  problems  can  be  tackled. 

The  main  difficulty  encountered  by  this  approach  for  solving  linear  programs  appears 
to  be  the  difficulty  of  knowing  "a  priori"  if  the  perturbed  quadratic  is  a  "proper* 
(according  to  the  results  of  [1]  and  [2])  perturbation  of  the  original  linear  program. 

The  main  result  of  this  paper  tries  to  overcome  this  difficulty,  in  fact  it  gives  an 
effective  computational  criterion  to  check  whether  the  solutions  of  the  perturbed  quadratic 
programs  provide  the  least  2-norm  solution  of  the  original  linear  program.  We  describe 
this  result  in  section  3.  In  section  2  we  give  a  new  convergent  result  of  a  sparsity 
preserving  snR  algorithm  (proposed  in  (3))  for  the  solution  of  a  class  of  quadratic 
programming  problems. 

In  section  4  we  present  some  algorithms  for  solving  the  linear  problem  with  their 
convergent  results.  We  briefly  describe  the  notation  used.  All  the  matrices  and  vectors 
are  real.  For  the  m  x  n  matrix  A  we  denote  row  1  by  A^,  column  j  by  A, j  and  the 
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element  in  row  1  and  column  j  by  For  x  In  the  real  n-dlmenslonal  Euclidean  apace 

denotes  element  1  for  1  >•  1,...,n  and  x^  denotes  the  vector  with  components 


(x^)^  •  max(x£,0}  1  ~  1,...,n.  All  vectors  are  column  vectors  unless  transposed  by  T. 
Ixl  will  denote  the  2-norm,  (x\)%  *  (  "  K  "onnegative 

n  ’ 

orthant  {x  :  x  e  R”,  x  >  0} .  For  a  point  c  in  R*'  and  a  closed  set  X  in  r'^  the  2 
norm  projection  pjCc.X)  of  the  point  c  on  X  is  defined  by 

•c  -  p,{c,X)l  «  min  Ic-xl  . 


s 


. 


2.  SOR  ALGORITHM  FOR  A  CLASS  OF  QUADRATIC  PROGRAMMING  PROBLRMS 
He  consider  the  following  separable  quadratic  prograit 


IT  T 

Min  y  X  DK  t  c  X 


s.t.  Ax  <  b 


where  D  is  a  positive  diagonal  matrix  in  r'**",  a  «  R**”,  c  e  r",  b  e  H*  and 

X>{x  >  Ax<b,  x>0}^i>  Associated  with  this  quadratic  program  is  the  dual  quadratic 


program  [4] 


IT  T 

Max  -  -j  X  Ox  -  b  u 


.t.  Dx  +  c  +  Au-v“0 


(u,v)  >  0 


which  upon  elimination  of  x  by  using  the  constraint  relation 

M  ■  -D"'(a‘*^u-v+c) 


IT  T  •  1  T  T 

Min  —  (A  u-v+o)  D  (A  u-v+c)  +  b  u 


s.t.  (u,v)  >  0  . 

Problem  (3)  can  be  solved  by  using  a  sparsity-preserving  SOR  algorithm  introduced  in  [3]. 
More  specifically  we  have  the  following  algorithm  tdiere  we  have  assumed  that  Aj  ^  0, 

Vj  -  1,...,m. 


QPSOR  algorithm 

Choose  (u°,v®)  e  R®*",  ui  t  (0,2). 

Having  (u'',v'')  compute  (u^'^^v''*^ )  as  follows: 


..  .  ■  .  •-  -V.  V-V- ■ 


.  --.  V 

•  ■>  V 


•V"  . 

.•.A':. 


s'*  V ' 
•  •  • • 


vV.-.V 


^  ^  a  w.wvw»j»-i 


».  .  s  %r.-.  .rw  . 


- -  (*4°"’(  I  <**>  4“t^'  +  I  (A*)  ju][  -  v’'  +  c)  +  *>4)) 4. 

'  '  .a.d-’/2,2  ^  1-1^  •“  l-J  •"  ^ 


J  for  1>1 

v*'’*'  -  (v’'  -  u(-a’’u’''^’  +  v’'  -  e))^  . 


J  -  1,...,» 


RBHARK.  Ih*  algorithm  works  with  the  rows  of  A  only  and  is  well  suited  for  aatrices  A 
which  have  a  pronounced  row  structure.  We  refer  to  [3]  for  a  similar  algorithm  which  works 
with  the  columns  of  A. 

Some  convergence  theorems  of  the  preceding  algorithm  were  given  In  [3]  and  [5] .  Here 
we  give  a  new  convergence  result  under  a  mild  regularity  assumption  on  the  constraints. 

PROFOSITIOH  1 .  Let  the  gradients  of  the  active  constraints  of  problem  ( 1 )  at  the  optliuil 
point  X  be  linearly  Independent.  Then 

a)  The  sequence  ((u^«v^)}  generated  by  QPSOR  algorithm  converges  to  a  point  (u,v)  which 
solves  problem  (3)  and  the  corresponding  x,  detenalned  by  (2)/  Is  the  unique  solution  of 
problem  ( 1 ) . 

b)  If,  moreover,  the  strict  complementarity  holds  at  the  optimal  point  x  of  (1)  (that 

Is  u^  >  0  If  Aj^x  •  bj^  and  v^  >  0  If  x^  >  0)  then,  there  exists  an  k  such  that  for 
k  >  k  the  QPSOR  algorithm  becomes 


ulf^’  -  0  ,  J  €  I 


“J 

v^'*’’  -  0  ,  j  e  I 


J-1 


“j*'  "  “l - - (a  d"’(  I  (a'').juJ*’  +  I  -  v’'  +  c)  +  b  ))  ,  J  e  I. 

V^'*'’  -  (v^  -  u(-a'Ju’'''‘’  v^  -  Cj))  ,  1  «  Ij 


where 


1,  -  {1  :  Aj^  X  <  b^)  ,  Ij  «  {1  :  Xj^  >  0} 


1^  -  {1  :  Aj^x  -  bj^}  ,  Ij  -  {1  I  Xj^  -  0} 


-4- 


and  tha  aaquanca  {u’^fv’^}  convargas  to  tha  optimal  point  (u,v)  at  llnaar  root  rate. 

PROOF 

a)  The  problem  ( 1 >  has  a  unique  solution  x  and  by  the  linear  independence  assumption  of 
the  gradients  of  the  active  constraints  it  has  a  unique  optimal  multiplier  (u,v)  ««hich  is 
also  the  unique  solution  of  problem  (3).  Now  from  part  iii)  of  'nteorem  1  of  [3]  it  follows 
that  the  {u*^,v^)  is  bounded  and  every  accumulation  point  solves  problem  (3).  But  problem 
(3)  has  a  unique  solution  and,  therefore,  the  bounded  sequence  {u^v'')  has  only  one 
accumulation  point  and,  hence,  it  converges. 

b)  The  proof  of  part  b]  follows  easily  by  repeating  the  same  steps  of  the  proof  of  nieorem 

2.5  of  t6]  by  taking  into  account,  that  from  part  a)  we  have  that  the  entire  sequence 
converges  and  by  noting  that,  from  the  linear  independence  assumption  of  the  gradients  of 
the  active  constraints,  we  obtain  that,  also  in  this  case,  the  matrix  (see  page  481 

of  [6] )  is  nonaingular. 

□ 

REMARK.  Part  b)  of  the  preceding  proposition  says  that  the  algorithm,  after  a  finite 
number  of  steps.  Identifies  which  variables  will  be  sero  at  the  solution. 


3.  PERTURBATION  OF  LINEAR  PROGRAMS 


We  consider  the  linear  program 

T 

Max  c  X 

s.t.  Ax  <  b 
X  >  0 

whore  A  e  r“*",  c  «  r",  b  e  rf*  and  X  »  {x  :  Ax  <  b,  x  >  0}  ^  ^ .  Let  X  denote  the 
(possibly  empty)  optimal  solution  set  of  (4).  First  of  all  we  recall  the  following 
fundamental  result  given  In  [1]. 

THEOREM  1.  Let  the  linear  program  (4)  be  feasible.  Then 

a)  1)  max  c'^x  has  a  solution  — >  le*  >  0  s  “  PjtO.jT)  for  all  e  e  (0,6*1 

xcx 

max  has  a  solution  | 
xex  I 

11)  _  f  <““  le*  >  0,  X*  :  “  **  ®  *  (0,6*1 

and  X*  =  P2(0,X)  1 

where  P2(x,X)  denotes  the  2-norm  projection  of  x  on  X. 

b)  sup  c^x  «  «  <««>  lp2(^,X)l  +  <•  as  e  ♦  O'*”. 

XfX 

PROOF.  See  [1]. 

c 

We  can  note  that  p2(^'*^  also  a  solution  of  the  problem 

6  T  T 
Min  —  X  X  -  c  X 

s.t.  Ax  <  b 
X  >  0 

and  that  the  quadratic  programming  dual  [3]  to  (S)  is 

IT  2  T 

Min  Y  lA  u  -  V  -  cl  +  cb  u 

s.t.  (u,v)  >  0 

where  the  primal  and  dual  variable  x  and  (u,v)  are  related  by 


1  T 

X-—  (-XU+V+-C) 


(7) 


Unfortunately  the  parameter  e*  In  Theorem  1  is  not  easy  to  compute  (see  [2])  and 
therefore,  in  9eneral,  we  can  not  be  sure  that  the  point  x(e)  obtained  by  solving  (5)  or 
(6)  is  the  optimal  solution  of  the  linear  program  (4).  Kven  if  we  repeat  the  computation 


of  points  P,(-^,X)  for  decreasing  values  of  until  the  condition  p-(  - 

c  c  ^ 

p^C— ,X)  is  verified  we  can  not  conclude  that  the  point  p^l^^fX)  Is  the  optimal  solution 


of  the  linear  problem  (4)- 

The  following  theorem  allows  to  overcome  this  last  difficulty  and  it  will  be  useful  in 
the  next  section  where  some  algorithms  for  solving  the  linear  program  (4)  will  be  proposed. 


THEOREM  2.  Assume  that  X  ^  d  and  let  the  gradients  of  the  active  constraints  of  the 
linear  program  (4)  at  the  optimal  point  x*  •  p^CO.X)  be  linearly  Independent. 

Let  (x,u,v)  and  (x,u,v)  be  two  points  that  satisfy  the  XXT  conditions  for 
problem  (5)  with  two  different  values  for  e,  namely  e  •  e  and  e  >  e  respectively 
where  e  -  0e  with  6  e  (0,1).  If  x  -  x  «  x»  then  it  follows: 

u  >  9u  ,  V  >  9v 


^x*  -  Pj(0,X^  <== 


T  .  .T,u-9u\ 

.0  X*  -  b 


Furthermore,  if  x*  «  Pj(0,X),  ("I™,  is  the  optimal  solution  of  the  dual  of  the 


linear  program  (4). 


PROOF 

(<--) 


Since  (x*,u,v)  and  (x*,u,v)  satisfy  the  KKT  conditions  for  (5)  with  c  and  e  we 


have 


«w 

Ex*  -  ctAu-v»0 


—  7^  — 

EX*  -  c  +  Au-v«0 


u'^(Ax*  -  b)  -  0 


(8) 

(8) 

(10) 


u’^lAx*  -  b)  -  0 


(11) 


(u.v)  >  0 


(u,v)  >  0  • 

By  taking  into  account  that  e  •  e/6,  from  (8)  it  follows 


£  ^ 

—  x*-e  +  Aa-v»0 


6 


which  Impllaa 

e  X*  -  8c  +  A*6u  -  8v  ”  0 
Now,  by  subtracting  (16)  from  (9)  and  by  dividing  by  (1-6)  we  obtain 


(16: 


,T,u-6u.  v-6v 


-  7:5-  -  0  • 


(17; 


By  using  (10)  -  (13)  we  have 


(^)^AX*-b)  -  0 


(is: 


^v-9v-(^ 

(-TTr-J  X*  -  0  . 


(is: 


U'^U. 


Therefore  from  (17),  (18),  (19),  (14),  (15)  and  the  nonnegativity  assumption  on  (-^^^ — ) 


V— Ov  ,\i*6u  v*6v 

and  ( )  it  follows  that  the  pair  ( yjy  ■  ,  is  dual  feasible  and  that  x*  is 


primal  feasible  for  the  linear  problem  (4)*  By  assumption  we  have  also  that 


c'^’x* 


b^(^) 


'1-6 


which  Implies  that  x*  Is  an  optimal  solution  for  the  linear  program  (4)  and  that 


,u-6u  v-6y, 

1-8 


(7:^  ,  )  Is  optimal  for  the  dual  of  the  linear  program  (4). 


Moreover  we  can  observe  that  both  (x*,  3)  and  (x*,  — )  satisfy  the  KKT 

e  e  e  e  e  e 

conditions  for  the  problem 


B>t.  kx  <  b 
X  >  0 


cx  >  Y 


where  y  is  the  maximum  value  of  (4),  from  which  It  follows  that  x  •  x  -x*  •  P2(0,X). 

Now  from  the  linear  Independence  assumptions  of  the  gradients  of  the  active  constraints  we 

have  that  x*  has  a  unique  optimal  multiplier  and,  therefore,  we  can  conclude  that 
u— 9u  V“6v 

( )  la  the  unique  solution  of  the  dual  of  (4).  In  fact.  If  It  had  a  different 

A  A  A  A 

solution  (u,v),  the  pair  (u,v)  should  be  another  optimal  multiplier  for  point  x*  (sei 
[4]  cap.  5  and  8). 

(-->) 


Por  simplicity  we  can  Introduce  the  vector  A,  A,  A* 

u 


I  A 


u  u* 

~  »  A*  - 

_  V  L  . 


where  (u*,v*)  are  the  KKT  multipliers  of  the  linear  problem  associated  to  the  optimal 


point  X*  "  P2(0,X) . 


Under  the  assumption  that  the  gradients  of  the  active  constraints  are  linearly 
independent  we  can  use  the  multiplier  function  A<x)  (see  [7]  and  [8])  to  compute  the 
values  of  A,  A  and  A*  in  function  of  x* 


A  (x*)  -  -D(x»)"^B''(ex»-c) 


A  (x«)  -  -D(x*)"’B''‘’(ex*-c]  -  -D(x*)"’b’'i|- X*  -  c) 


A*  -  A*(x*)  -  -D(x«)“’b'’^[-c1 


(20 

(21 

(22 


where 


A 

Diag  (A.x*  - 
1<l<m 

bj^l  0 

A 

■> 

Diag  lA.x*  -  b.J  0 

KKm 

-I 

0 

Diag  txj) 
KKn  - 

-I 

0  Diag  (x*J 

KKn  J 

and 

B  -  [A^,  -1]  . 

From  (20)  -  (22)  it  follows 


u-6u  I 

1-9  I 

v-6v 
.  1-6 


u* 


V* 


y— Oyx  _ 

namely  t)ie  pair  [  y-g—  ,  is  t)ie  optimal  KKT  multiplier  of  the  linear  problem  (4). 

□ 

REMARK,  i)  From  the  proof  of  the  Theorem  we  can  note  that  the  regularity  assumption  on  the 
active  constraints  is  not  needed  for  the  implication  (<"). 

From  Theorem  1  and  Theorem  2  we  can  also  state  the  following  results i 

COROLLARY  1 .  Assume  that  X  ^  f)  and  let  the  gradients  of  the  active  constraints  of  the 
linear  program  (4)  at  the  optimal  point  x'  «  p^lO.X)  be  linearly  independent.  Then  there 
exist  an  e*  >  0  such  that  for  all  e  €  (0,e*]  we  have 

u(e)  -  u*  +  ey 


v(e)  “V*  ey 


wh*r«  (tt*,v*)  is  ths  optlMl  solution  of  ths  dual  of  ths  llnsar  program  (4)  and 
(u(e),v(c))  Is  ths  unlgua  solution  of  (6),  and  y  is  a  vsctor  indspandsnt  of  e. 

PROOF,  ths  proof  follows  from  Thsoram  1  and  (20)  -  (22). 

o 

Ths  nsxt  rssult  givss  a  particular  eharactsrisation  of  ths  solvability  of  a  llnsar 
program. 

COROUARY  2.  Lst  (x.UiV)  and  (x,u«v)  bs  two  points  that  satisfy  ths  nCT  conditions  for 
problem  (5)  with  two  different  values  for  e,  nasMly  c  •  c  and  e  •  c  respectively 
where  e  •  6c  with  6  t  (0,1). 
i)  Suppose  that 
a)  X  «  X  •  X* 

of  problem  (5)  are  linearly  independent  at 


X  I*  I)  and  X*  ■  P2(0,X). 

(4)  is  solvable  and  the  gradients  of  the 
optimal  point  x*  ■  pjlOfX)  are  linearly 
such  that  the  conditions  a)  -  d)  of  part 

1)  hold. 

PROOF.  The  proof  of  part  1)  follows  by  repeating  the  same  steps  of  the  first  part  of  proof 
of  Theorem  2. 


b)  nte  gradients  of  the  active  constraints 
the  points  x* 

c)  u  >  6u,  V  >  9v 

T  .  v.'r,u-9u, 

d)  c*x»  »  b  ( ■> 

then  the  linear  program  (4)  is  solvable  (namely 
li)  Conversely,  suppose  that  the  linear  program 
active  constraints  of  linear  program  (4)  at  the 
independent,  then  there  exist  two  values  for  c 


The  proof  of  part  il)  follows  from  Theorem  1  and  Theorem  2 


4.  SOR  ALGORITHM  FOR  LIHEAR  PROGRAMMING 


I 

( 

The  quadratic  prograomlng  problem  (6)  can  be  solved  by  a  sparsity-preserving  algorithm 
which  follows  directly  from  the  QPSOR  algorithm  of  section  3  by  replacing  D  by  el  (also 
In  this  case  we  have  assumed  that  Aj  ^  0  Vj  «  1,...,m). 

LPSOR  algorithm 

Choose  (u°,v®)  €  R^!*",  ui  e  (0,2)  and  e  >  0. 

Having  (u’',v'')  compute  (u*'*'\v*''*’^)  as  follows 


k+1  /  k  “  k  ?  ,.T,  k  k  ,  .  .V 

1  ■  IT?  1  ill  ilj  ^  ^ 

^  j>i 


j  “  i» 


k+1  ,  k  ,  .T  k+1  ^  k  ^  .. 

V  -  (V  -  (<i(-A  u  +  V  +  c)) 


By  using  the  results  of  sections  2  and  3  we  can  state  the  following  convergence  result 
which  sharpens  previous  LPSOR  convergence  results  given  in  [1]  and  [S] > 

PROPOSITION  2.  Assume  that  X  0  and  let  the  gradients  of  the  active  constraint  of  the 
linear  program  (4)  at  the  optimal  point  x*  •  p2(0,X)  be  linearly  Independent.  Then 

a)  There  exists  a  real  positive  number  e*  such  that  for  each  e  e  (0,e*],  the  sequence 
{(u’^,v^))  generated  by  the  LPSOR  algorithm  converges  to  a  point  (u(e),v(e))  which  solves 
problem  (6)  and  the  corresponding  x(e)  determined  by  (7)  Is  Independent  of  e  and 

x(e)  -  x«  -  p^lQ.X). 

b)  If,  moreover,  the  strict  complementarity  holds  at  the  optimal  point  x*  •  P2(0,X)  then 
there  exists  a  real  positive  number  e**,  c**  <  c*,  such  that  for  each  e  e  (0,e**]  there 
exists  an  k  such  that  for  k  >  k  the  LPSOR  algorithm  becomes 

Uj'*'’  -  0  .  1  *  ^1 

-  0  ,  j  e  Ij 


,.k+1 


.A^r 


^  1-1 
j>i 


,  T.  k+1 
(A  ).jUj  + 


m 

j. 

t-j 


(A*>.juf 


-  c)  +  eb^. 


i  «  1, 
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vf  ’  -  ^  Cj)  .  J  * 

tfhara 

I,  -  (i  »  A^x*  <  b^}  ,  Ij  -  {1  I  X*  >  0> 

-  {1  1  A^x*  -  bp  ,  Ij  -  (1  I  X*  -  0} 

and  the  aaquanea  {(u^tV^)}  eonvargas  to  tha  optiaal  point  (u(e),v(e))  at  linear  root 
rate. 

PROOF 

a)  Zt  follows  directly  frda  Proposition  1  and  ntaoram  1. 

b)  If  (u(e),v(e))  is  a  solution  of  problast  (6)  for  a  fixed  e,  e  <  e*,  we  have  by  using 
Corollary  1 

lin  (u(e).v{e))  -  (u*,w*) 

€♦0 

ifhere  (u*fV*)  is  the  nCT  multiplier  of  tha  linear  problem  (4). 

Mow  by  using  the  strict  complementarity  assumptions  and  theorem  1,  we  have  that  there 
exists  an  e**,  e**  <  e*,  such  that  for  all  e  «  (0,e**] 

{1  »  Uj^(e)  >  0}  ■  {i  I  uj  >  0)  ,  {1  j  v^(e)  >  0)  ■  {1  »  vj  >  0) 

{i  I  u^(e)  -  0}  ■  {i  I  uj  -  0)  ,  {it  Vj^(e)  -  0}  »  {1  I  vj  -  0> 

Therefore «  by  taking  into  account  that  e**  <  e*  and  Theorem  1,  it  results  that  the  strict 
complementarity  assumption  holds  also  for  the  problem  (5)  for  all  e  e  (0,e**]  and,  again 
the  result  follows  from  Proposition  1. 

( 

In  applying  part  i)  of  Proposition  2  we  must  be  able  to  select  a  value  of  e  such 
that  e  <  e*.  In  order  to  ensure  that  e  <  e*  we  may  have  to  choose  very  small  values 
for  e.  Computational  results  have  shown  that  very  small  values  for  e  yield  a  very  slow 
convergence  of  the  LP80R  algorithm  when  applied  to  the  problem  (6). 

In  order  to  overcome  the  lack  of  a  practical  "a  priori"  selection  procedure  for  the 
parameter  e*  we  can  propone  two  algorithms  Which  are  based  on  the  results  of  the 


preceding  section 


ALGORITHM  I 


Choose  6  e  (0>1),  and  >  0. 

tat  -  (e)^e°,  1  -  0,1,2,...  and  let  (u(e^>,v:(e^) )  b«  a  solution  of  problaa  (6) 
with  e  -  e^,  and  x(e^)  defined  by  (7)  with  e  -  and  u  ■  u(e^),  v  •  w(e^). 

By  using  Theorem  1  and  Theorem  2  we  can  give  the  following  convergence  result  for 
algorithm  I. 

PROPOSITICM  3.  Assume  that  X  ^  0  and  let  the  gradients  of  the  active  constraints  of  the 
linear  program  (4)  at  the  optimal  point  x*  •  P2(0,X)  be  linearly  independent.  Aien  for 
some  integer  i 

xle^)  -  x(e^"’)  -  X*  -  P2(0,X) 

is  optimal  for  the  linear  program  (4)  and 

ru(e^)  -  eu(e^"’)  v<€^)  -  6v(e^"’)> 

^  1-e  '  J 

is  the  optimal  solution  of  the  dual  of  the  linear  program  (4). 

The  next  algorithm  contains  an  automatic  adjustment  rule  for  the  parameter  e  based 
on  the  abstract  models  considered  in  [8].  It  carries  out  the  minimisation  of  two  problem 
(6)  for  t%io  different  values  of  e  and  uses  the  obtained  points  to  adjust  the  value  of  e 
during  the  minimisation. 

In  the  sequel  %fe  assume  that  we  have  an  algorithm  defined  by  an  iteration  map 
F  !  X  r"  X  R  ♦  2*^  *r"^  such  that  for  any  fixed  value  of  e,  any  accumulation  point 

of  the  sequence  produced  by  F  gives  a  point  x(e)  and  a  pair  (u(e),v(e))  which  solve 
the  problems  (5)  and  (6)  respectively  for  the  given  e. 

For  simplicity  we  introduce 

B  -  [a'',-I1,  M  -  b'^B,  d  -  C^),  q  -  (q),  A  -  (")  , 

where  B  e  M  «  ,  d,  q,  A  e  and,  in  the  sequel,  we  denote  the 

scalar  product  with  xy.  The  algorithm  model  described  below  makes  use  of  two  preselected 


••qa«ne«a 

and  {s’*} 

Data  t 
Stap  0: 
Stap  It 
Stap  2t 
Stap  3 I 
Stap  4 I 

Stap  Si 

Stap  6i 

Step  7t 

Stap  Si 

Stap  9 I 
Stap  10 1 


J ■-■aval' 


{el>  and  {H**),  \rtiara  0  <  <  Oe^  <  el,  j  ■  ,  {el>  +  0  aa  J  ♦  • 

♦  •  aa  k  ♦ 

kLSORITIM  IX 

9  €  (0,1),  k^,k2,k3,k4,kj  >  0,  (X®,x**,X®,x®) . 

Sat  k  •  0,  j  •  0. 

Sat  i  -  0  and  (X^,x^,r‘',*^)  -  (x’',x'',X*,?‘). 

Computa  (X^‘*'',x^‘*'')  «  r(X^,x^,el),  (X^*^,x^*S  e  F(r^,x^,9e^)  and  aat  i  ■  i+1 
If  1  <  go  to  atap  2,  alaa  go  to  atap  4. 

Sat  -jJ-  -  Xi  -  (X^  -  (MX^  +  d  +  elq))+,  5^  -  X*  -  (X^  -  (MX^  +  d  ♦  9e^q))^. 

If  IT  I  •  0,  IT  I  *  0  go  to  atap  S,  alaa  go  to  atap  6* 

If  Ix^  -  x^l  ■  0,  (X*  -  9X^)  >  0  and  Ic^Cx*’  -  9x^)  -  b'^(u*'  -  9u^)|  ■  0  aat 

,,k+1  k+1  rk+l  He+1 ,  i  rd.  — i.  .  ...  . 

(X  ,x  ,X  ,x  )  •  (X  X  ,X  ,x  ),  k  ••  k'4-1  atopi 

alaa  go  to  atap  9. 

If  K,(|X^(!li^+  q)|  +  lx^(‘£^*q)|)  *  (kx^  -  b)^  ♦  -  b)^) 

e^  Oe-* 

•*•  Kj((-x^)^  +  {-x*’)^)  >  -  x^l*  go  to  atap  7»  alaa  go  to  atap  9. 

If  K^<ec^  I  Ix^l^  -  Ix^l^l  +  |9e^<lx^l^  -  IxS*)  -  e^Cx^  -  9x^) 

-t-  b’c^  -  9a^)l)  >  e^|c''(x*'  -  9x^)  -  b’^ci?  -  eu^)|  go  to  atap  8i 

alaa  go  to  atap  9. 

If  lCj(lT^I  -*•  IT^I)  >  (9X^ 
alaa  go  to  atap  9, 

Sat  J  -  j+1. 

Sat  (X  ,x  ,X  ,x  )  “  (X  ,x  ,X  ,x  ),  J(k+1)  «  J,  k  ■  k+1  and  go 

to  atap  1> 


Tha  algorithm  II  la  akatchad  in  figure  1 


THEOKBM  3.  Aaauaa  that! 

a)  nia  linaar  prograa  (4)  la  aoXvabla* 

b)  nia  gradianta  of  the  active  oonatrainta  are  linearly  Independent  at  the  optimal  point 
«'  “  P2(0,X). 

c)  The  atrlct  compleaMntarlty  holda  at  the  optiaMl  point  x'  p^iOiX). 

d)  For  a  fixed  e  the  Iteration  map  F  la  conatltuted  by  an  iteration  of  the  ifSOR 
algorithm  and  by  the  relation  (7). 

Than^  i)  aither  tha  aljgorithm  tarminataa  *t  aooa  (X  ,x  )  whara  x  *  x  ■  p2(0«X) 

la  optimal  for  the  linear  program  (4)  and  ( ~  •  )  the  optimal  aolutlon  of  the  dual  of 

the  linear  program  (4)i 

11)  or  producea  Infinite  aequencea  and  {j()c))  auch  that  (jOc)}  la 

bounded  (namely  la  changed  only  a  finite  number  of  tlmea),  {x^)  and  {x*^  converge 

to  the  optimal  aolutlon  x'  •  p_(0,X)  of  the  linear  problem  (4)  and  the  aequence 
-4c  k  ^ 

{ '}  convergea  to  tha  optlaial  aolutlon  of  the  dual  of  the  linear  program  (4). 

PROOF.  Firat  of  all  we  prove  that  the  aequence  {j(k)>  la  bounded. 

We  prove  thla  by  contradiction.  Therefore  we  auppoae  that  the  aequence  {j(k)}  la 
unbounded  and  conalder  the  teata  at  atep  5.  atep  6«  atep  7  and  atep  8  that  ahould  have 
Increaaed  the  value  of  parameter  j  at  atep  9. 

Let  e*  >  0  be  the  nundber  conaldered  In  Theorem  1  then  we  muat  have  for  aome  k*. 
gj(k)  g  ^21  k  >  k*.  By  ualng  Theorem  1  and  Theorem  2  It  followa  that  the 

algorithm  could  not  have  Increaaed  the  value  of  parameter  j  on  account  of  a  failure  to 

aatlafy  the  teat  in  atep  5.  In  fact  if  II*'  -  (A**  -  {Ml*'  +  d  e^*''*q))^l  •  0  and 

IX*'  -  (T*'  -  (mT**  +  d  +  9e^^*'^q))^l  •  0  for  k  >  k*  we  have  that  X*'  and  X*  aolve  (aae 

t6]  page  472)  problem  (6)  with  e  ••  and  c  •  reapectlvely.  Then  Theorem  1 

impliea  that  li^  -  x*'l  >0  and  Theorem  2  that 

(I*  -  ex*')  >  0  and  lc<x*'  -  ex*')  -  b(^  -  9u’')l  -  0 
and  hence,  the  algorithm  ahould  have  terminated  at  atep  5.  Now  we  conalder  the  teat  at 
atep  6  for  k  >  k*.  If  we  denote  with  x(e),  u(e),  v(e)  the  aolutlona  of  problema  (5)  and 


(6),  It  follows  from  Lenma  2.1  of  [10]  that 


lx’'  -  (  U’'(— j-~-  ♦  qll  +  lu(e^‘’'’)l  I  (Ax’'  -  b)^l 


.1?'  -  X(et3<’'>,.2  <  ^_(|x*(a^,  KA?.  -  b),. 


ee-i 


♦  lv(6e^*’'’)l  K-x’')^! 


Since  for  k  >  k*  (and,  hence,  <  e*)  we  have  that 


x(cJ<'‘>)  -  x(ee^*’'’)  -  X*  -  P2(0,X) 


by  using  (23)  and  (24)  we  obtain  for  k  >  k* 


(23 


(24 


+  q1l  +  qlh  +  Kj(l(Ax’'  -  b)^l  +  I  (Ax'  -  b)^l  )  + 


+  Kj(l(-x’')^l  +  l(-l?')^l)  -  (e^**'’)^!!?'  -  x’'l^  >  (k,  -  -t-  q)  I 


(’'1 


cl*’*’  -1,  wT*  *  a 

-5—)  ^ 


Oe 


j(k) 


+  q)l  +  (Xj  -  e^*’'^u(c^*’'’ )l  )l  (Ax’'  -  b)^l 


J(k) 


+  (kj  -  — — lu(0e^‘’'’)l)MAx'  -  b)^l  +  (kj  -  e^‘’'’lv(e^‘’'’)l  )l(-x’')^l 

,j(h) 


t  (K3- 


—  lv(0e^^''’)l)l(-5^)^l 


From  assumption  b)  it  follows  that  {u(e^*’'’)},  {u(6e^*’'’ )}  ,  {v(e^'’'’)),  {v($e^^’'’)}  are 
bounded  for  all  k  >  k*  and  therefore  we  can  observe  that  the  test  at  step  6  Is  satisfied 
for  sufficiently  large  values  of  k  (and  hence  for  sufficiently  small  values  of 
The  same  arguments  hold  for  test  at  step  7,  in  fact  we  have 

K^(9e^‘’'’|ll^l^  -  lx''l^|  +  l0e^*''’(lx''l^  -  lx’'l*)  -  c(x’'  -  Ox’')  +  b(S’'  -  eu’')|) 

-  e^*’'’|c(x’'  -  Ox*')  -  b(^u'  -  eu’')|  >  (K.  -  e^‘’'’)|c(x’'  -  Ox’')  -  b(u'  -  0u’')| 


and,  hanea,  tha  taat  la  aatlafiad  for  auffiaclantly  larga  valuaa  of  k  (hanca  for 


auffieiantly  aaall  valuaa  of 

Finally  «a  varify  tha  taat  at  atap  8.  Wa  can  nota  that,  for  auffieiantly  larga  valuaa 
of  k,  (and,  hanca,  for  auffieiantly  larqa  valuaa  of  and  auffieiantly  amall  valuaa 

of  Thaoraa  1,  Corollary  1,  Propoaition  1  and  Propoaltion  2  imply 

j  -  {i  ,  1*  -  0)  -  {1  I  -  0)  -  {i  1  -  0)  -  {1  »  -  0)  -  {i  I  -  0) 

(2 

I  -  (i  «  X*  >  0}  -  {i  1  x^  >  0}  -  (i  I  x^  >  o>  -  {i  I  x^i*^*’'*)  >  0}  -  {i  I  x^(ee^*'‘’)  >  0} 

whara  X*  •  (tt*,v*)  ia  tha  XXT  aultipliar  of  tha  linaar  problam  (4)  and  • 

(uCe^**'' )  ,v(el**‘* ) )  and  X(0e^^*'*)  •  ,v(ea^^*'* ) )  ara,  again,  tha  aolution  of 

problaa  (6)  with  e  ■  and  e  «  0*^*'''  raapactlvaly. 

Naxt  wa  hava  VI  <  I 

-  0X^  -  (1-0)X«  ♦  (^  -  Xj^(0e^‘''’))  *  (X^Cde^**'^  -  X*)  -  0(X^  -  X^(e^‘''*)) 

-  0(Xj^(e^“‘’)  -  XJ)  >  |(1-0)X*^|  -  |X^  -  X^(e«^‘''>)l  -  IX^Oe^*’'*)  -  X*| 

-  0|x][  -  X^(f^<''’)|  -  0|X^(e^‘’'’)  -  X*1  . 


Than  by 

conatruction  of  {x'*} 

and 

(X*> 

and  by 

uaing  that  lim  •  • 

and 

lim  c 

k-*« 

k-M* 

lin 

■ax{|X^  -  X  (ee^‘'‘’>|. 

|X  (0e^'’'’ 

in  atap  2  and  atap  3,  by  racalling  Corollary 
“  0  wa  obtain 

)  -  hi  -  X^{e^‘’'’)|,  |x^(e^“'’)  -  Xjl)  . 


1 


0 


and,  for  auffieiantly  larga  valuaa  of  k, 

X*  -  01^  >  0  Vi  e  I 
i  X 

and,  by  taking  into  account  (25), 

I*  -  ©X**  >  0  . 

Hanca,  wa  hava  that,  for  auffieiantly  larga  valuaa  of  k,  tha  taat  at  atap  8  ia  aatiafiad. 

Tharafora  wa  can  concluda  that  j  can  not  ba  incraaaad  an  inf ini ta  nuabar  of  timaa  aa 
hypothaaisad,  but  thara  axlata  a  k**  auch  that  j(k)  «  j*  for  all  k  >  k*». 


Mow  if  th*  algoritha  torminatas  at  atap  S  part  i)  of  tha  niaoraa  follows  from  ihaoraa 


1  and  ■niaoraa  2. 

If  tha  algoritha  producaa  Inflnita  saquancaa  (1^)  and  than,  by  Proposition  2, 

*  ^ 

thasa  saquancaa  convarga  to  two  points  1*  and  X*  which  solve  tha  problea  (6)  with  two 
different  values  of  e  (that  is  €  -  e^*  and  e  -  Be^*)  and  x*  and  x*  solva  tha 
priaal  problea  ( 5 ) . 

nierafore  %n  have 


q)|  -  0  . 


(AX*  -  b)^  -  0  , 


q)|  -  0  . 


(Ax«  -  b)^  -  0  , 


(26) 

(27) 


(-x*)^  -  0  , 


(-x*)^  -  0  , 


(X*  -  (MX*  +  d  +  e^*q))^l  -  0  ,  iT*  -  (T*  -  (mT*  d  ♦  0e^*q))J  -  0  , 


(28) 

(29) 


and  froa  equality  battiaan  the  priaal  objective  function  and  the  dual  objective  function  of 
problea  (5)  it  follows! 


»  2  T* 

lx*l  -  C  X* 


lx*l*  -  b*^*. 


•)*  *  2  T*  T* 

lx*l  -  c  X*  +  b  u»  -  0  , 


6e^  —  2  T—  6e^*  —  2 

~ —  lx*l  -  c^  -  -  —  lx*l 

•I*  _  2  T-  T— 
ee-*  lx*l  -  c  X*  b  u*  -  0 


b  u*  , 


Now  by  ( 26 ) ,  ( 27 ) ,  ( 28 )  and  step  6  we  have 

lx*  -  x«l  «  0  .  (31) 

Froa  (31),  (30)  and  step  7  it  results 

|(1-e)c''x*  -  b^(u*  -  eu*)|  -  0  . 

By  (29)  and  step  8  we  obtain  also 

u*  >  0u*  ,  V*  >  6v* 

so  that  the  second  part  ii)  of  Theorea  follows,  again,  froa  llieorea  1  and  Theorea  2> 

Q 
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